The current group behavior recognition method do not make full use of the group relational information, so that the group recognition accuracy cannot be effectively improved. Therefore, a deep neural network model based on the hierarchical relational module of Affinity Propagation (AP) algorithm was proposed, named Clustering Relational Network (CRN). First, Convolutional Neural Network (CNN) was used to extract scene features, and the regional feature clustering was used to extract person features in the scene. Second, the hierarchical relational network module of AP was adopted to extract group relational information. Finally, the individual feature sequences and group relational information were fused by Long Short-Term Memory (LSTM) network, and the final group recognition result was obtained. Compared with the Multi-Stream Convolutional Neural Network (MSCNN), CRN has the recognition accuracy improved by 5.39 and 3.33 percentage points on Volleyball dataset and Collective Activity dataset, respectively. Compared with the Confidence-Energy Recurrent Network (CERN), CRN has the recognition accuracy improved by 8.70 and 3.14 percentage points on Volleyball dataset and Collective dataset, respectively. Experimental results show that CRN has higher recognition accuracy in the group behavior recognition tasks.